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Introduction 


Systems analysis develops and documents candidate mission and architectures, associated system 
concepts, enabling capabilities and investment strategies to achieve NASA’s strategic objectives. The 
technology assessment process connects the mission and architectures to the investment strategies. In 
order to successfully implement a technology assessment, there is a need to collect, manipulate, analyze, 
document, and disseminate technology-related information. Information must be collected and organized 
on the wide variety of potentially applicable technologies, including: previous research results, key 
technical parameters and characteristics, technology readiness levels, relationships to other technologies, 
costs, and potential barriers and risks. This information must be manipulated to facilitate planning and 
documentation. An assessment is included of the programmatic and technical risks associated with each 
technology task as well as potential risk mitigation plans. Risks are assessed and tracked in terms of 
likelihood of the risk occurring and consequences of the risk if it does occur. The risk assessments take 
into account cost, schedule, and technical risk dimensions. Assessment data must be simplified for 
presentation to decision makers. The Systems Analysis and Concepts Directorate (SACD) at NASA 

Langley Research Center has a wealth of experience in performing Technology Assessment and Portfolio 

i 

Analysis as this has been a business line since 1978. 

Systems and Technology Engineering 

In order to design a system to meet an operational need, assumptions must be made about the state 
of the technology base applicable to all components of the system. In order to select and develop 
technologies in a cost-effective manner, assumptions must be made about the level of benefit that the 
technology will have to the system or systems in which it would be inserted and employed. Any complete 
systems engineering framework must include an integrated methodology that addresses engineering and 
management issues related to the concurrent selection, development and insertion of advanced 
technologies. 

Technology engineering is a distinct discipline, separate from, but highly related to, systems 
engineering. Technology engineering and systems engineering are usually performed by different 
organizations, often with different cultures, including employees and managers with different personalities 
and educational backgrounds. Figure 1 depicts Technology Synergy Teams (TSTs) comprised of 
technologists and subsystems analysis and Systems Synergy Teams (SSTs) made up of systems engineers 
and analysis. Technology engineering begins with bottoms-up innovation, often unrelated to any particular 
system or product, and ends with technology insertion prior to system development. Systems engineering 
begins with a product focus and does not end until the product is fielded. In technology engineering, 
requirements are treated more as goals, with a given technology project having a high degree of uncertainty 
of success at achieving the desired performance on schedule; hence, multiple alternative technology 
projects are typically managed in parallel within a larger technology program, in order to have fail-back 
positions to reduce risk. Roadmaps are typically constructed with decision gates to reduce alternatives as 
progress is made or to introduce new or modified technology projects as required. In systems engineering, 
the focus is on designing and developing a product to meet the requirements on schedule and cost. 


1 

American Institute of Aeronautics and Astronautics 



AIAA 2006-7029 



Technology Synergy Teams (TSTs) 


System Synergy Teams (SSTs) 



TechnotogisB and Subsystem Analysis 


System Engineers and Analysts 


TSF-ttS 


4 


Figure 1 - Technology and System Synergy Teams 


The engineering of systems and the technology required to make them successful must often be 
performed concurrently, with a very high degree of interaction and communication. Any framework for 
technology engineering must draw on existing systems engineering processes, tools, and methods; take into 
account the differences between systems engineering and technology engineering; and account for the high 
degree of interaction required between the concurrent technology and system development programs. 


A technology engineering framework has been developed for the selection, development, and 
insertion of technologies concurrent with the systems engineering of the system(s) in which the technology 
will eventually be employed. Figure 2 provides an iterative waterfall flow of the key steps in this 
technology engineering framework, which include: (1) technology requirements development and analysis, 
(2) technology prioritization and initial selection, (3) technology program planning, (4) technology program 
monitoring and management, (5) and technology integration and insertion. 

Mission Need and System Requirements : As indicated on the top side of Figure 2, the product- focused 
process begins with a mission need. This mission need is translated into a system need and associated 
requirements and eventually into a technology need and associated requirements. This top-down system 
requirements analysis and allocation process follows standard systems engineering practices; including 
functional analysis and allocation, systems analysis and trade studies, system design integration, etc. 2 3 

Technology Requirements Development and Analysis : The first major element of the technology 
engineering framework in Figure 2 is "Technology Requirements Development and Analysis". This is an 
iterative process consisting of bottoms-up and top-down components. Technology requirements come from 
a top-down system requirements analysis and allocation approach. System- and subsystem-level 
requirements are translated into technology’ requirements through the use of Technology Sensitivity 
Derivatives (TSDs) and Technology Technical Performance Measures (TTPMs). A TSD is the partial 
derivative of the change in a system-level requirement (e.g. life-cycle cost, reliability) to a change in TTPM 


Systems and Technology Engineering Framework 


2 

American Institute of Aeronautics and Astronautics 










AIAA 2006-7029 


(e.g. thermal conductivity, combustion efficiency). A TTPM is a subsystem- or component-level parameter 
that is readily measurable during the implementation of a given technology project by a knowledgeable 
technologist. 


Systems Engineering and Technology 
Engineering Framework Interactions 
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System 

Development 
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Figure 2 - Systems Engineering and Technology Engineering Framework 

The initial set of top-down technology requirements generated through this process are then 
validated or iterated through a bottoms-up technology identification process involving expert technologists. 
As shown in the bottom portion of figure 3, promising technologies that have the potential of meeting the 
identified requirements are identified from the stream of on-going technology basic research. In some 
cases technologies may be identified that have the potential to exceed the identified requirements, and in 
other cases, it may not be possible to identify a portfolio of technologies that can meet the requirements in 
the given timeframe. The technologist can determine which technologies have the highest potential for 
improvement, and the systems analyst or engineer can determine which technologies have the largest 
potential impact on the system. Hence, an iterative, collaborative process between technologists and 
systems analysts and engineers must be executed to identify from the broad pool of promising technologies 
a smaller set of screened technologies that have the potential to meet or exceed the iterated technology 
requirements. 

For this set of screened technologies, a wide variety of documentation should be developed, 
including: previous research status and results, key technical details, related TTPMs, current and required 
Technology Readiness Levels (TRLs), cost and schedule (including milestones) to achieve the required 
TRLs, an assessment of the risk of achieving the required TRLs, interrelationships with other technologies, 
and related TSDs and potential system-level benefits. These details should be documented in a database. 

This set of screened technologies and documentation will then be used to accomplish the next 
major element of the technology engineering framework. It should be noted that this initial screening and 
documentation of promising technologies could be considered as a high-level initial implementation of the 
first three elements in figure 2, to be followed by a more detailed iteration (e.g. a spiral model). 
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Figure 3 - Technology Prioritization Approach 
Technology Prioritization and Initial Selection 


The process takes the screened technologies and associated documentation as shown in figure 3, 
together with the technology requirements and Technology Sensitivity Derivatives (TSDs), and performs 
technology prioritization and initial selection to develop a technology investment portfolio for program 
planning and execution. The prioritization occurs against evaluation criteria derived from the system-level 
requirements in cooperation with the customer. The prioritization process takes advantage of TSDs, which 
can be used to quantitatively evaluate the integrated impact of employing a given technology advancement 
on the system-level Figures of Merit (FOMs). 

Although much quantitative data can be made available to assist in the evaluation process, some 
portion of the evaluation and prioritization process will almost always have to be qualitative in nature 
because of the large number of technologies and FOMs, the limitations of existing systems analysis models, 

and the need to often include qualitative evaluation criteria. Any tools and methods used to facilitate this 

2 

process must take these factors into account. 

A large number of processes, tools, and methods have been proposed and demonstrated in recent 

3 , 4,5 

years for multi-attribute decision making. Some of these approaches have been demonstrated to a 

6 , 7,8 

limited degree for the specific application of technology prioritization. These approaches include the 

9 10 11 

use of the Analytic Flierarchy Process, Quality Function Deployment, Multi- Attribute Utility Theory -- 
among others. Some of the approaches include methods for dealing with probabilistic risk and 

5 , 6 , 7, 8 

uncertainty. Many of these processes could be adapted and expanded for this application; however, the 
approach used in this framework should have the following characteristics: 

(1) Make maximum use of quantitative data; 

(2) Include the capability to perform qualitative and quantitative evaluations; 

(3) Allow collaborative, real-time participation by experts and stakeholders; 
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(4) Provide a method for weighting evaluation criteria or combining them into a utility function; 

(5) Allow the evaluation of costs as well as benefits; 

(6) Allow the integrated evaluation of technologies against multiple systems; 

(7) Include consideration of incompatibilities or interactions between technologies; 

(8) Include the capability to assess uncertainty and quantify risk; 

(9) Allow the performance of sensitivity analyses and visualization of results; 

(10) Be well-founded in mathematical theory; 

(1 1) Be easy to understand and explain; and 

(12) Be systematic, repeatable, objective and open to scrutiny. 

It is important in the process of managing complex aerospace programs to understand the 
technical, schedule, and cost risks of each technology required to achieve the goals of the program. In 
addition to the internal program assessments of risks, it is prudent to have independent assessments of risk 
conducted by qualified experts outside of the day-to-day program. Conducted properly, these independent 
assessments can provide unbiased insight to program management to support technical and programmatic 
planning and decision-making. The objective of technology development risk assessment is to use all 
available program information to estimate the risk that each of the technologies under development will not 
be sufficiently mature to meet system level requirements within the time and fiscal constraints of the 
program. The assessment addresses both technical and programmatic (schedule and cost) risks. These 
assessments are conducted independent of any internal program risk management processes that might be 
in use. 
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Figure 4 - Technology Investment Portfolio Development 

The output of this process is a portfolio of prioritized technologies, see figure 4, for potential 
investment and inclusion into an integrated technology development program. This process can be kept in 
a database format and continually updated as required by changes in requirements and FOMs, system 
designs, technology program progress, etc. The selected portfolio of technologies is then used to provide a 
recommended prioritized technology budget profile. 

Examples of Langley SACD technology assessment and portfolio analysis experiences are given 


5 

American Institute of Aeronautics and Astronautics 





AIAA 2006-7029 


below: 


High-Speed Research (HSR) Program (1990-1999) 

The HSR technology tracking and assessment process was the process by which NASA and its 
industry partners measured the technical progress of the program’s technology development efforts. 
Metrics were established for each technology area and tracked against end-of-program projections. The 
process was probabilistic in nature and consisted of two phases. The technology audit phase was where the 
data was gathered and compared to previous assessments. The metrics integration phase was where 
uncertainties in system-level FOMs of merit are quantified by using a Monte-Carlo approach to propagate 
uncertainties at the technology level. 

The objective of the NASA HSR program was to develop the technologies (see figure 5) necessary 
to design, develop, manufacture, field, and support a High Speed Civil Transport that was both 
economically viable and environmentally acceptable. Phase I of the HSR Program successfully identified 
technologies that mitigate environmental concerns. The focus in Phase II was on technologies in the areas 
of aerodynamics, propulsion, structures and materials, and flight deck systems that address issues of 
economic viability. 


The benefits from research and development expenditures on technology advancement and 
subsequent synthesis of the resultant technologies into a final application or product are not always 
obvious. During the program's conceptual phase, when many promising technologies were being identified 
and explored, the end-results of a specific technology development effort were difficult to forecast. 
Technology breakthroughs and failures will occur. The nature and frequency of these occurrences were 
challenging to predict with any degree of certainty. Therefore, one aspect of technology development 
program management should be to acknowledge the presence of uncertainty, measure and track it, and, if 

li 

possible, reduce the uncertainty to an acceptable level . 
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Figure 5 - High Speed Research Critical Technologies 
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Evaluating Technology Impacts on Mission Success of Future Launch Vehicles (1999-2002) 

A process was developed for tracking focused technology development by integrating system and 
uncertainty analyses. Uncertainties were developed for single-stage and two-stage reusable launch 
concepts from historical weight estimating relationships that are typically used during the concept 
definition phase of a system. Other uncertainties were developed based on empirical data and NASA 
guidelines. The uncertainties were integrated into the systems analysis model, and a Monte Carlo 
simulation was conducted to determine the integrated uncertainty probability. The results (see figure 6) 
showed there is a 31 percent weight growth (4.3 payloads) uncertainty in dry weight between a 50 percent 
and 95 percent probability for the Single-Stage-to-Orbit concept. The Bimese concept (two identical 
Single-Stage concepts mated and sized for two-stage performance) resulted in a reduction in both weight 
(12 percent) and uncertainty (15 percent or 1.8 payloads). These uncertainties are similar to the NASA 

guideline for weight growth of spacecraft. Finally, the present process was extended as a model for 

12,16 

measuring the progress of technology development programs. 


Reusable Launch Vehicle (RLV) 
Prioritization Results (1999-2002) 
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Figure 6 - Reusable Launch Vehicle Prioritization Results 
Capability Requirements Analysis and Integration (CRAI) (2003-2005) 

The CRAI team was chartered (see figure 7) to identify and prioritize the capabilities and technologies 
that must be developed before the Nation’s Space Vision can be attained. Reporting to the Space Architect, 
the CRAI team was organized around nine critical capability areas to accomplish this goal. In keeping with 
the One NASA philosophy, membership of the CRAI Team was drawn from across the National 
Aeronautics and Space Administration (NASA). An Independent Technology Assessment Team (IT AT) 
was also formed to assess the data generated by the CRAI Team, to prioritize capabilities, and to identify 
capability gaps. 

In addition to nuclear propulsion and power capabilities, the following are CRAI examples of critical 
capabilities for which significant advances beyond the current state-of-the-art (SOA) will be needed to 
pursue the President’s vision. Prioritization of the technologies required to enable these capabilities will be 
heavily influenced by the specifics of design reference missions and architectures and the launch capability 
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to LEO. 

• High reliability computing systems 

• integrated Systems Health Management (ISHM) 

• Thermal protection systems 

• In-space propulsion 

• Extra- Vehicular Activity (EVA) suits 

• Closed-loop life support systems 

• In-space storage, transfer, and management of cryogenic fluids 

• Surface/sub-surface mobility 

• Radiation and Micro-meteorite Orbital Degree (MMOD) protection 

• Lightweight thermal management 


The CRAI process produced a very large, well documented, and easily referenced database of 

13 

capabilities and technologies. 


Capability Requirements, Analysis, & 
Integration (CRAI) Concept (2003-2005) 
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Figure 7 - Capability Requirements, Analysis, & Integration Concept 

Vehicle Systems Program (VSP) portfolio analysis (2004-2005) 

An assessment was performed to project the potential benefits of Vehicle Systems Program (VSP) 
portfolio of advanced technologies on a fleet of subsonic transports. This activity was a joint effort 
involving analysts from LaRC’s Systems Analysis Branch (SAB) and GRC’s Aerospace Systems Analysis 
Office (ASAO), in concert with Georgia Tech’s Aerospace Systems Design Laboratory (GT/ASDL). This 

nd 

task was the 2 of three planned FY05 analyses agreed to by VSP’s Vehicle Integration, Strategy and 
Technology Assessment (VISTA), the Subsonic Transport (ST) Vehicle Sector Manager and the NASA 
systems analysis organizations (SAB & ASAO). 


8 

American Institute of Aeronautics and Astronautics 







AIAA 2006-7029 


The approach to performing this analysis (see figure 8) consists of resizing a set of subsonic 
transport reference vehicles to take advantage of the potential benefits of VSP technologies. The resulting 
advanced vehicle characteristics are then compared to the reference vehicles in order to measure the 
potential progress towards the Goals/Objectives/Capabilities of the Subsonic Transport Vehicle Sector. 

14 

Advanced vehicles were analyzed for a 5-year and a 15-year timeframe . 

The Subsonic Transport Sector Analysis Team has completed evaluation of the applicable current 
VSP technology portfolio against the Subsonic Transport Sector goals and critical 15-year capabilities. The 
analysis was performed on four separate vehicles including a 70 passenger regional jet, a 150 passenger 
mid-range transport, a 300 passenger long-range transport, and a 300 passenger long range Blended-Wing- 
Body transport. The results indicate that Empty Weight, SFC, Engine TAV and LTO NO long term goals 

X 

may be achievable. However, little or no progress is being made towards achieving the L/D goal. The 
Noise Reduction community is considering revising the metric used for community noise, which, in turn, 
would lead to a more realistic target goal value. Results clearly demonstrate the need for a rigorous 
systems analysis approach since technologies focused on improving one goal often have negative impacts 
on multiple other goals/capabilities. Also, the interdependencies of different goals can produce counter 
productive results as indicated by the L/D vs. Empty Weight dilemma. One final point is that for this 
analysis a current TRL is associated with each technology analyzed, hence one should have more 
confidence in the potential benefits of the higher TRL technologies. 

15 

The VSP Return on Investment (ROI) calculator was created to allow the Vehicle System 
Analysts and decision makers to quantitatively visualize the benefit to the commercial aviation industry 
from the VSP technology roadmaps. Each metric was changed from its normal unit of measure into a 
monetary benefit through an individualized conversion. To make an accurate assessment of the impact on 
the commercial aviation system, a fleet growth model and a user-defined advanced vehicle introduction rate 
were used to model the air transportation system of the future. These components were brought together to 
quantify the overall return on investment from the VSP portfolio. 


Vehicle Systems Program (2004-2005) 
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Figure 8 - Vehicle Systems Program 
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Exploration Systems Architecture Study (ESAS) (2005) 

In May of 2005, NASA initiated the Exploration Systems Architecture Study (ESAS). The purpose of 
the study was to: 

1) Assess the top-level Crew Exploration Vehicle (CEV) requirements and plans that enable the CEV 
to provide crew transport to the International Space Station (ISS); 

2) Define top-level requirements and configurations for crew and cargo launch systems to support the 
lunar and Mars exploration programs; 

3) Develop a reference exploration architecture concept to support sustained human and robotic lunar 
exploration operations; and 

4) Identify key technologies required to enable and significantly enhance these reference exploration 
systems and a reprioritization of near-term and far-term technology investments. 

SACD had significant roles in supporting the ESAS study team. SACD performed the liaison risk 
assessment functions between the ESAS team and an agency wide team charged with using the Shuttle to 
complete the ISS by 2010. SACD performed numerous Lunar Surface Access Module (LSAM) trades to 
define top level element requirements and establish architecture propellant needs. The technology 
assessment process (see figure 9) was developed and implemented by SACD as the ESAS architecture was 
refined. 
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Figure 9 - ESAS Technology Assessment Overview 
Summary 

This paper has developed and presented a technology engineering framework for the selection, 
development, and insertion of technologies concurrent with the systems engineering of the system(s) in 
which the technology will eventually be employed. The technology engineering framework developed and 
presented in this paper draws on existing systems engineering processes, tools, and methods; takes into 
account the differences between systems engineering and technology engineering; and accounts for the 
high degree of interaction required between the concurrent technology and system development programs. 
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The major steps in the technology engineering process are: (1) technology requirements development and 
analysis, (2) technology prioritization and initial selection, (3) technology program planning, (4) 
technology program monitoring and management, (5) and technology integration and insertion. Process, 
tools and personnel exist with SACD to perform technology assessment and portfolio analysis studies for 
the Agency either in direct support to a program and/or in an honest broker fashion. 
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